eBPF: Kernel Observability Without Recompiling

javi
Pantalla con métricas de sistema y gráficos en tiempo real

eBPF (extended Berkeley Packet Filter) is one of the most transformative technologies of recent years in Linux, yet it remains unknown outside specialised circles. It’s the foundation many modern observability, networking, and security tools build on — Cilium, Pixie, Falco, Tetragon, and others. We cover what it is, why it matters, and where you’ll find it (probably without knowing) in your stack.

The Core Idea

Traditionally, modifying Linux kernel behaviour required:

  • Compiling a module and loading it (risk: a bug = kernel panic).
  • Patching the kernel and recompiling (impractical in production).
  • Using userspace utilities that only saw what the kernel exposed (limited).

eBPF changes this. It lets you load small programs in the kernel that execute in response to events (syscalls, network packets, file accesses, process changes). These programs:

  • Load dynamically — no recompilation or reboot required.
  • Are statically verified before executing — the kernel verifier rejects programs that could hang it, access disallowed memory, or loop endlessly.
  • Run in a virtual machine inside the kernel, with restricted types and limited stack.
  • Are JIT-compiled to native code for performance.

Result: you can instrument the kernel in real time, with safety similar to userspace, and performance close to native code.

Why It Matters

Before eBPF, observing what happened inside the kernel required heavy tools (SystemTap, DTrace in its limited Linux versions) or instrumentation that significantly slowed the system. eBPF changed that equation: fine observability with small overhead.

Use cases that open up:

  • Syscall and process tracing with no extra agent or reboots.
  • Packet filtering without going through iptables/nftables (better performance).
  • Visibility into encrypted traffic by observing before/after encryption in the kernel.
  • Runtime security auditing with details auditd doesn’t capture.
  • Application profiling with function-level stack resolution without instrumenting the binary.

Tools Built on eBPF

In 2023 there’s a notable ecosystem of products that depend entirely on eBPF:

  • Cilium: Kubernetes networking with eBPF instead of iptables. Better performance and observability (Hubble) in the same package.
  • Pixie: Kubernetes observability capturing HTTP, mySQL, DNS, etc. without instrumenting apps.
  • Falco: runtime threat detection. Originally with kernel module; migrated to eBPF for greater compatibility.
  • Tetragon: runtime security enforcement with declarative policies.
  • bcc/bpftrace: libraries and tools to write eBPF programs more easily.
  • Parca: continuous low-overhead profiling of production processes.

If you use Cilium or Pixie, you’re already using eBPF whether you know it or not.

A bpftrace Example

To see eBPF “live”, bpftrace is the simplest entry. Count syscalls per process in one line:

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

This loads an eBPF program that fires on every syscall, indexes by process name (comm), and increments a counter. Prints results on Ctrl-C. Without touching app code, without restarting anything.

Other useful one-liners:

  • read() latency per process: bpftrace -e 'kprobe:vfs_read { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns[comm] = hist(nsecs - @start[tid]); delete(@start[tid]); }'
  • TCP retransmissions: bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[comm] = count(); }'

Restrictions Worth Knowing

eBPF is not magic and has limits:

  • Strict verifier. The verifier rejects programs it considers unsafe even if they would work. Sometimes you have to rewrite logic to pass verification.
  • No free recursion. Loops are allowed in modern kernels but with bounded iterations.
  • Limited stack (512 bytes). You can’t have large structures.
  • Kernel compatibility. Advanced features require relatively new kernels. RHEL 7 has very limited support; RHEL 9, Debian 12, Ubuntu 22.04 are reasonable.
  • Learning curve. Writing eBPF directly is complex. Most of us use bpftrace or frameworks like bcc.

How to Start Learning

If you want to dig deeper than “eBPF is good”:

  • Brendan Gregg has the best public resources on tracing and eBPF — his book BPF Performance Tools (2019) remains current.
  • ebpf.io — centralised site with tutorials, ecosystem, and references.
  • Practical lab: install bpftrace on your Linux, try the documentation one-liners, then write your own.
  • For serious programming: learn Rust or C + libbpf-rs / Aya framework.

Conclusion

eBPF has redefined what’s possible to observe and modify on a modern Linux system. It’s beneath many modern Kubernetes and observability tools — understanding at least the concepts helps you decide between alternatives and debug when those tools behave unexpectedly. You don’t have to write eBPF directly to benefit, but knowing it’s there expands your repertoire.

Follow us on jacar.es for more on observability, Linux kernel, and modern Cloud Native tools.

Entradas relacionadas